Modern computing and display technologies have facilitated the development of systems for so called “virtual reality” (“VR”), “augmented reality” (“AR”), and/or “mixed reality” (“MR”) experiences. This can be done by presenting computer-generated imagery to a user through a head-mounted display. This imagery creates a sensory experience which immerses the user in a simulated environment. VR systems typically involve presentation of digital or virtual image information without transparency to actual real-world visual input.
AR systems generally supplement a real-world environment with simulated elements. For example, AR systems may provide a user with a view of a surrounding real-world environment via a head-mounted display. Computer-generated imagery can also be presented on the head-mounted display to enhance the surrounding real-world environment. This computer-generated imagery can include elements which are contextually-related to the surrounding real-world environment. Such elements can include simulated text, images, objects, and the like. MR systems also introduce simulated objects into a real-world environment, but these objects typically feature a greater degree of interactivity than in AR systems.
Various optical systems generate images at various depths for displaying VR/AR/MR scenarios. Some such optical systems are described in U.S. Utility patent application Ser. No. 14/555,585 and Ser. No. 14/738,877, the contents of which have been previously incorporated by reference herein.
AR/MR scenarios often include presentation of virtual image elements in relationship to real-world objects. For example, referring to FIG. 1, an AR/MR scene 100 is depicted wherein a user of an/a AR/MR technology sees a real-world park-like setting 102 featuring people, trees, buildings in the background, and a concrete platform 104. In addition to these items, the user of the AR/MR technology perceives that they “see” a robot statue 106 standing upon the real-world platform 104, and a cartoon-like avatar character 108 flying by which seems to be a personification of a bumble bee, even though the robot statue 106 and the cartoon-like avatar character 108 do not exist in the real-world environment. While FIG. 1 schematically depicts an/a AR/MR scenario, the quality of the AR/MR scenario varies depending on the quality of the AR/MR system. FIG. 1 does not depict a prior art AR/MR scenario, but rather an AR/MR scenario according to an embodiment.
The visualization center of the brain gains valuable perception information from the motion of both eyes and components thereof relative to each other. Vergence movements (i.e., rolling movements of the pupils toward or away from each other to converge the lines of sight of the eyes to fixate upon an object) of the two eyes relative to each other are closely associated with accommodation (or focusing) of the lenses of the eyes. Under normal conditions, accommodating the eyes, or changing the focus of the lenses of the eyes, to focus upon an object at a different distance will automatically cause a matching change in vergence to the same distance, under a relationship known as the “accommodation-vergence reflex.” Likewise, a change in vergence will trigger a matching change in accommodation, under normal conditions. Working against this reflex, as do most conventional stereoscopic VR/AR/MR configurations, is known to produce eye fatigue, headaches, or other forms of discomfort in users.
Stereoscopic wearable glasses generally feature two displays—one for the left eye and one for the right eye—that are configured to display images with slightly different element presentation such that a three-dimensional perspective is perceived by the human visual system. Such configurations have been found to be uncomfortable for many users due to a mismatch between vergence and accommodation (“vergence-accommodation conflict”) which must be overcome to perceive the images in three dimensions. Indeed, some users are not able to tolerate stereoscopic configurations. These limitations apply to VR, AR, and MR systems. Accordingly, most conventional VR/AR/MR systems are not optimally suited for presenting a rich, binocular, three-dimensional experience in a manner that will be comfortable and maximally useful to the user, in part because prior systems fail to address some of the fundamental aspects of the human perception system, including the vergence-accommodation conflict.
VR/AR/MR systems such as the ones described in U.S. Utility patent application Ser. No. 14/555,585 address the vergence-accommodation conflict by projecting light at the eyes of a user using one or more light-guiding optical elements such that the light and images rendered by the light appear to originate from multiple depth planes. The light-guiding optical elements are designed to in-couple virtual light corresponding to digital or virtual objects, propagate it by total internal reflection (“TIR”), and then out-couple the virtual light to display the virtual objects to the user's eyes. In AR/MR systems, the light-guiding optical elements are also designed be transparent to light from (e.g., reflecting off of) actual real-world objects. Therefore, portions of the light-guiding optical elements are designed to reflect virtual light for propagation via TIR while being transparent to real-world light from real-world objects in AR/MR systems.
To implement multiple light-guiding optical element systems, light from one or more sources must be controllably distributed to each of the light-guiding optical element systems. The light is encoded with virtual image data that is rendered at a relatively high rate (e.g., 360 Hz or 1 KHz) to provide a realistic 3-D experience. Current graphics processing units (“GPUs”) operating (e.g., rendering virtual content) at such speeds and at a high resolution consume a large amount of power (relative to the capacity of a portable battery) and generate heat that may be uncomfortable for a user wearing the AR/MR system.
AR/MR scenarios often include interactions between virtual objects and a real-world physical environment (e.g., the robot statue 106 standing upon the real-world platform 104 in FIG. 1). Similarly, some VR scenarios include interactions between completely virtual objects and other virtual objects.
Delineating surfaces in the physical environment facilitates interactions with virtual objects by defining the metes and bounds of those interactions (e.g., by defining the extent of a particular surface in the physical environment). For instance, if an AR/MR scenario includes a virtual object (e.g., a tentacle or a fist) extending from a particular surface in the physical environment, defining the extent of the surface allows the AR/MR system to present a more realistic AR/MR scenario. In one embodiment, if the extent of the surface is not defined or inaccurately defined, the virtual object may appear to extend partially or entirely from midair adjacent the surface instead of from the surface. In another embodiment, if an AR/MR scenario includes a virtual character walking on a particular horizontal surface in a physical environment, inaccurately defining the extent of the surface may result in the virtual character appearing to walk off of the surface without falling, and instead floating in midair.
To facilitate interactions between virtual objects and real-world physical environment, various AR/MR systems utilize fiducial markers (see ArUco markers 200 of FIG. 2) to provide position and orientation (i.e., pose) information for real-world physical surfaces on which the fiducial markers are placed. However, ArUco markers 200 do not provide any information relating to the extent of a physical surface. Moreover, few applications or situations are amenable to the placement of ArUco 200 markers on one or more surfaces in a real-world physical environment. For instance, ArUco markers 200 can alter the aesthetics of a surface by requiring a visible marker to be placed on that surface.
While some VR/AR/MR systems can generate polygon meshes to delineate and/or represent surfaces in the physical environment, such polygon meshes may provide too much information for facilitating interactions between virtual objects and real-world physical environment use. For instance, a VR/AR/MR system would need to further process polygon meshes for various applications/functions/processes such as simulating physical collisions, simulating resting contact, and various lighting effects (e.g., shadows and reflections). Further processing of polygon meshes for these various applications/functions/processes with sufficient speed and resolution to enable a realistic, believable and/or passable VR/AR/MR experience can require many processor cycles. Processor related requirements may in turn impose performance (e.g., processor cycles for other functions such as rendering), power (e.g., battery life), heat (e.g., in view of proximity to user's body), and size (e.g., portability) related restrictions on VR/AR/MR systems. There exists a need for more abstract and easily digestible representations of the environment to represent key aspects of the environment, such as the location of large flat regions with minimal processing. Polygon meshes require further processing to abstract out useful information. The systems and methods described herein are configured to address these and other challenges.